Skip to content

MCP Issue Labeler GitHub Triage Bot#13393

Draft
anannya03 wants to merge 5 commits intomainfrom
add-mcp-labeler
Draft

MCP Issue Labeler GitHub Triage Bot#13393
anannya03 wants to merge 5 commits intomainfrom
add-mcp-labeler

Conversation

@anannya03
Copy link
Member

@anannya03 anannya03 commented Dec 18, 2025 •
edited
Loading

MCP Issue Labeler GitHub Triage Bot

This PR adds an issue labeling system for the microsoft/mcp repository using Azure AI Search, Azure OpenAI, and RAG (Retrieval-Augmented Generation).

Changes

MCP Labeler (McpOpenAiLabeler)

  • Automatically predicts Server and Tool labels for new MCP issues
  • Uses RAG to retrieve 15 similar labeled issues for context
  • Applies rule-based reasoning with confidence thresholding (0.9)
  • Prevents data leakage via excludeIssueId filter
  • Achieves ~95% accuracy on evaluation dataset
image

MCP-Specific Infrastructure (McpTriageRag)

  • Query preprocessing to remove tool name examples (e.g., "like Storage")
  • Filters for labeled issues only (Server ne null)

Evaluation Tool (Mcp.Evaluator)

  • Console app for testing labeler accuracy against ground truth
  • Single issue prediction, batch evaluation, and test dataset extraction modes
  • Exports results to CSV with detailed metrics

@anannya03 anannya03 requested a review from joshfree January 5, 2026 05:10
@github-actions
Copy link

github-actions bot commented Mar 13, 2026

Hi @anannya03. Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days.

@github-actions github-actions bot added the no-recent-activity There has been no recent activity on this issue. label Mar 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@joshfree joshfree Awaiting requested review from joshfree

@jsquire jsquire Awaiting requested review from jsquire jsquire will be requested when the pull request is marked ready for review jsquire is a code owner

@jeo02 jeo02 Awaiting requested review from jeo02 jeo02 will be requested when the pull request is marked ready for review jeo02 is a code owner

@radhgupta radhgupta Awaiting requested review from radhgupta radhgupta will be requested when the pull request is marked ready for review radhgupta is a code owner

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

no-recent-activity There has been no recent activity on this issue.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@anannya03